Programme For International Student Assessment Data Research Articles

Overclaiming. An international investigation using PISA data

ABSTRACT This paper investigates the phenomena of overclaiming – the propensity for individuals to claim more knowledge about an issue or topic than they really (or could possibly) do. Using Programme for International Student Assessment (PISA) data from nine Anglophone countries and over 40,000 young people, we examine teenagers’ propensity to claim knowledge of three mathematics constructs that do not really exist. We find substantial differences in young people’s tendency to overclaim across countries, genders, and socio-economic groups. Those who are most likely to overclaim are also found to exhibit high levels of overconfidence and believe they work hard, persevere at tasks, and are popular amongst their peers. Together this provides important new insight into overclaiming, how this differs across groups, and how it relates to other psychological constructs.

Perceptions of key education actors towards PISA: the case of Scotland

ABSTRACT This study explores the perceptions of government officials, teachers, and parents in Scotland regarding the use of Programme for International Student Assessment (PISA) results to evaluate national education performance. International large-scale assessments (ILSAs) such as PISA have been increasingly influencing education policymaking worldwide, but there is limited understanding of how education actors perceive these assessments. This study uses Scotland’s recent declining performance in PISA after the implementation of the Curriculum for Excellence (CfE) reform in 2010 as a case study. Interviews with key officials involved in implementing the CfE, and parents’ and teachers’ representatives reveal their doubts about the validity of PISA as a means of assessing Scottish education performance. This is because PISA data lack comparability and accountability, as students’ academic performance is not impacted by PISA scores and teacher evaluations are not based on them. They do not consider that the PISA results reflect the effects of the CfE reform. These views contradict the Scottish government policy that PISA, among other assessments, is meant to inform policymakers and evaluate reform effects. Our findings have implications for countries that routinely use PISA to assess their educational performance and do not consider the views of education actors.

The learning gain over one school year among 15-year-olds: An international comparison based on PISA

We compare the learning gain over one year of schooling among 15-year-old students in Austria, Brazil, Malaysia, Scotland (United Kingdom) and Singapore. Common metrics for reading, mathematics and science learning, as established by the Programme for International Student Assessment (PISA), are used. In order to overcome the limitations of a cross-sectional, single-cohort design, we combine multiple vintages of PISA data and exploit the fact that the testing period in these countries varied over the years. The results show that students’ yearly learning progress around the age of 15 varies from about one-tenth of a standard deviation in students’ test scores in Malaysia to about one-fourth of a standard deviation or more in Austria and Scotland.

Reading instruction and reading engagement and their relationship with Chinese students’ PISA reading performance: Evidence from B-S-J-Z, Hong Kong, and Chinese Taipei

Chinese students’ reading performance in the Programme for International Student Assessment (PISA) has attracted worldwide attention. Using 2018 PISA data from Beijing-Shanghai-Jiangsu-Zhejiang, Hong Kong, and Chinese Taipei (n = 25,338), we explored whether the relationship between reading instruction, engagement, and performance differs across the three Chinese regions. Results revealed that disciplinary climate, teacher support, and cognitive engagement had a significant positive relationship, while teacher-directed instruction and student-perceived reading difficulty had a significant negative relationship with reading performance in all three regions. Distinct patterns were identified in the relationships between student-perceived teacher stimulation, teacher feedback, self-concepts of reading competence and reading performance across the three regions. The findings highlight the importance of considering intra-cultural diversity when studying Chinese educational practices.

Mathematics, Gender and the Meaning in Life: The Results of PISA testing in Bosnia and Herzegovina

<p style="text-align: justify;">The Program for International Student Assessment (PISA) was developed by the Organization for Economic Cooperation and Development to measure students’ knowledge and skills needed for today’s society. PISA is a large-scale assessment of 15-year-old students in reading, mathematics, and science. In this analysis of PISA data from Bosnia and Herzegovina (BIH), we examined the relationship between gender, mathematics achievement, and perceived meaning in life in BIH students. The sample for this analysis comprised 6480 students (3148 females and 3332 males). The results of the analysis revealed a small but statistically significant, negative relationship between mathematics and the student’s perception of the meaning in life. Boys achieved higher scores in mathematics than girls, but the difference was relatively small. In addition, boys’ rating of meaning in life was higher than that of girls. Knowing what factors influence mathematical achievement might help educators create better intervention programs. In conclusion, we provided some possible explanations for these data.</p>

Analisis Skor Literasi Membaca Siswa Indonesia Menggunakan Linier Mixed Models

The linear mixed models is a development of the linear model which includes both fixed and random effects in the model. Random effect in the model is used to model complex data that has a grouping structure. The grouping structure can occur because the same observations are measured repeatedly or each observation is measured only once but these observations have some form of group structure. Students who participate in the Program for International Student Assessment (PISA) are nested in several schools, so the PISA data structure is quite complex and requires a more in-depth analysis. Quantitative studies on PISA, especially in reading literacy, are still rarely done. The purpose of this study is to determine what factors effect the Indonesian student’s PISA reading literacy scores using a linear mixed model approach with school being used as a random effect in the model. The findings of the study are that the factors that affects Indonesian student’s PISA reading literacy scores are the class being taken, gender, mother's highest education, facilities at home, school entry age, student discipline and failed a grade. The result of the estimation of random effect variance which is not equal to zero indicates that there is a random effect from the student’s school on PISA reading literacy scores. Based on model diagnostics and parameter testing, it was concluded that the model obtained is fitted in modeling Indonesian student’s PISA reading literacy scores.

Measuring family wealth among secondary school students with home possessions indicators: Investigation of dimensionality and measurement problems

This article investigates methodological problems of measuring family wealth among secondary school students with home possessions indicators. It employs multiple correspondence analysis (MCA) in order to explore (1) the dimensionality of home possessions items, (2) the nature of relationship between these items, and (3) the substantive meaning of the first and other relevant dimensions. It also employs regression analysis in order to study potential explanatory factors of the second dimension of the MCA solutions that may be related to measurement problems of the instrument. Two sources of data are used: (1) open access data from the National Survey of Student Achievement (NSSA) in Lithuania that provide a series of national educational studies conducted from 2002 to 2016, and (2) data from the Programme for International Student Assessment (PISA) study conducted in Lithuania in 2018 that is used to corroborate findings produced using the first data source. Results of the data analysis show that the first dimension of MCA solutions obtained from home possessions items in both NSSA and PISA data sets reflect high vs. low socio-economic status (SES) of the students. However, there is at least one more important dimension, and it reveals multi-dimensional nature of the measurement instrument. Analysis of potential explanatory factors of the second dimension of the MCA solutions revealed that reading abilities and school-related affect is related to this dimension. Thus, the second dimension of MCA solutions might also reflect methodological problems related to the measurement of home possessions in adolescent surveys.

From a crisis of results to a crisis of wellbeing – education reform and the declining sense of school belonging in Sweden

ABSTRACT In response to declining results in the Programme for International Student Assessment (PISA) surveys, the then governing Swedish coalition in 2010–2014 introduced earlier grading, more extensive national testing and a new standards-based curriculum. These reforms coincided with a greater emphasis on inclusive’ education understood in the ‘narrow’ sense of placement in mainstream schools. The combination of these two sets of reforms presents an interesting national case where traditional conservative demands for a core curriculum, testing and accountability were combined with calls to increase educational opportunity. Combining different methods and analysis of five separate waves of PISA data, we show that the reforms coincided with a decline in the sense of school belonging among pupils that was exceptional compared to other high-income countries, and especially among marginalised pupils. The study adds to the cumulative work of previous studies on policy effects on wellbeing, concluding that the Swedish compulsory school went from undergoing a mediatised results crisis to a mental health crisis among pupils.

Examining high achievement in mathematics and science among post-primary students in Ireland: a multilevel binary logistic regression analysis of PISA data

In Ireland, while, on average, students have performed well on national and international assessments of mathematics and science, the low proportions of high achievers in these subjects are noteworthy. Given these patterns and the multifaceted benefits in individual and societal terms that expertise in mathematics and science has been associated with, policymakers in Ireland have begun to place an increasing emphasis on high achievement in these subjects. This emphasis has coincided with ongoing efforts during the last decade to raise interest and improve academic performance within the realm of science, technology, engineering, and mathematics (STEM) education.Despite this policy attention, research on high achievement in mathematics and science nationally, but also internationally, has been particularly scarce. In an attempt to provide research evidence that could add further impetus to the ongoing efforts, this study examines high achievement in mathematics and science among post-primary students in Ireland using data from the 2012 and 2015 cycles of the Programme for International Student Assessment (PISA). Specifically, the study aimed to evaluate the contribution of various contextual characteristics stemming from students, their families, teachers, and schools in the prediction of high achievement in mathematics and science within a two-stage analysis that included a series of bivariate tests and multilevel binary logistic regression modelling.The results showed that variables related to students’ self-beliefs, engagement, and socioeconomic background were consistently associated with high achievement in mathematics and science. Overall, the significant role of students’ homes and families in predicting students’ chances of being high achievers in the two subjects was highlighted. In turn, this indicated that further efforts to enhance collaboration between teachers, schools, and parents may be warranted if progress in the area of high achievement in mathematics and science is to be made. The implications of these findings for policy and practice within the Irish context, the limitations of the study, and recommendations for future research are discussed.

Contrasting Profiles of Low-Performing Mathematics Students in Public and Private Schools in the Philippines: Insights from Machine Learning.

Filipino students performed poorly in the 2018 Programme for International Student Assessment (PISA) mathematics assessment, with more than 50% obtaining scores below the lowest proficiency level. Students from public schools also performed worse compared to their private school counterparts. We used machine learning approaches, specifically binary classification methods, to model the variables that best identified the poor performing students (below Level 1) vs. better performing students (Levels 1 to 6) using the PISA data from a nationally representative sample of 15-year-old Filipino students. We analyzed data from students in private and public schools separately. Several binary classification methods were applied, and the best classification model for both private and public school groups was the Random Forest classifier. The ten variables with the highest impact on the model were identified for the private and public school groups. Five variables were similarly important in the private and public school models. However, there were other distinct variables that relate to students’ motivations, family and school experiences that were important in identifying the poor performing students in each school type. The results are discussed in relation to the social and social cognitive experiences of students that relate to socioeconomic contexts that differ between public and private schools.

Does the material well-being at schools successfully compensate for socioeconomic disadvantages? Analysis of resilient schools in Sweden

BackgroundA variety of studies point to a deterioration of educational equity in Sweden and increasing school segregation with respect to achievement and socioeconomic composition. Some schools are resilient to socioeconomic disadvantages in their student body and demonstrate high levels of achievement. However, little attention has been given to these resilient schools. Material well-being, as one important dimension of student well-being, comprises the student’s home background and school resources. The relationship between home background and achievement is well-established but less literature includes school-level factors of material well-being. In comparing the material well-being at resilient, non-resilient, and more advantaged schools, this study aims at detecting possible patterns that may provide crucial information as to why some schools succeed better in compensating for disadvantages.MethodsUsing Swedish data from the Programme for International Student Assessment (PISA) from 2000 to 2018, the shares of resilient, non-resilient, and more advantaged school groups with different achievement levels were identified by using aggregated achievement and socioeconomic background measures. Making use of a well-being framework specifically designed for PISA data, the school groups were compared regarding their material well-being as measured by the perceived shortage of material resources and teachers, the percentage of teachers fully certified, the availability of computers, and extracurricular activities. This comparison of school groups was computed using the nonparametric Kruskal-Wallis test and a Bonferroni-adjusted pairwise comparison.ResultsThe shares of resilient schools decreased considerably from 14% in 2000 to 3% in 2015. Yet, the comparison of the material well-being at resilient and other school groups led to mostly non-significant results. Overall, disadvantaged schools reported higher teacher shortages than advantaged schools, which indicates the need for a more compensatory allocation of (human) resources.ConclusionsThe study concluded that the landscape of resilient schools is under continuous change. As no patterns of significant differences between resilient and other school groups were found, the study shows no indication that the material well-being at school compensates for disadvantages in a school’s student body. The findings call for further research regarding changes in the presence of resilient schools and their possible relationship with school material well-being.

A systematic review of studies investigating science teaching and learning: over two decades of TIMSS and PISA

ABSTRACT A great number of studies have investigated science teaching and learning (STL) using data from the Trends in International Mathematics and Science Study (TIMSS) and Programme for International Student Assessment (PISA). Nevertheless, there is little effort to synthesise these studies. Reviewing the status of research on STL, including the trends, approaches, and findings is crucial for identifying research gaps that require greater attention. Consequently, this review synthesises empirical studies investigating STL using TIMSS and PISA. First, we mapped their characteristics based on the aims, data, STL measures, and research approaches. Second, patterns of the findings were summarized by focusing on the (1) relationships between STL and student outcomes; (2) factors that explain STL variation; and (3) patterns of STL. Most studies examined STL related to inquiry activities and their relationships to student outcomes. Their findings were largely inconsistent and could be traced back to different ways STL was conceptualized and measured. This review calls for the studies examining TIMSS and PISA data to enhance the quality of research approaches and clarity in reporting them. It also encourages researchers to reflect upon the knowledge gained from harnessing these data to examine STL and discusses the challenges and opportunities that lie ahead.

RESTRICTED MAXIMUM LIKELIHOOD ESTIMATION FOR MULTIVARIATE LINEAR MIXED MODEL IN ANALYZING PISA DATA FOR INDONESIAN STUDENTS

The Program for International Student Assessment (PISA), becomes one of the references or indicators used to assess the development of students' knowledge and skills in each member country of the Organization for Economic Cooperation and Development (OECD). The results of the PISA survey in 2018 placed Indonesia in the bottom 10, indicating that the implementation of the national education system has not been successful. This underlies the need for a more in-depth study of the factors that influence PISA data scores not only statistically qualitatively but also quantitatively which is still very rarely done. The data structure of the PISA survey results is complex, which involves multicollinearity, multivariate response variables, and random effects. Thus, it requires an appropriate statistical analysis method such as the multivariate mixed linear regression (MLMM) model. In this study, secondary data from the results of the 2018 PISA survey with Indonesian students as the smallest unit of observation were used as sample. School is used as an intercept random effect which is assumed to be normally distributed. Multicollinearity is overcome by selecting independent variables based on AIC and BIC values. Estimation of variance and random effect parameters was performed using the restricted maximum likelihood (REML) method. Based on the estimator of the variance of random effects for the response variables of mathematics, science, and reading literacy, it was obtained 1548.12, 1359.39, and 1082.48, respectively, which explains the significant effect of each school as a random effect on the three response variables.

Panic and stoicism: Media, PISA and the construction of truth

In December 2019, the Organisation for Economic Co-operation and Development released the latest results of its triennial Programme for International Student Assessment (PISA) testing. What followed was a flurry of media reports in the participant countries about the 2018 PISA results that were interpretive rather than descriptive. Whilst there is a considerable body of literature describing PISA’s impact on education policies, there are fewer studies that examine how PISA data is presented to the public by the media and how this contributes to the construction of policy truths. As part of a broader study of policy landscapes in the Asia Pacific region, this paper provides a critical analysis of the mediatisation of the 2018 PISA results in Australia and Singapore. It will be demonstrated that in Australia, much of the reporting was characterised by panic, whereas in Singapore the reporting was much more measured, exhibiting a stoicism. Drawing on a corpus of articles published in Australian and Singaporean newspapers in December 2019, and informed by the work of Michel Foucault, the analysis will make visible how dominant discourses used by the media operated as regimes of truth to advance neoliberal imperatives.

The impact of test language on PISA scores. New evidence from Wales

AbstractIn this paper we investigate the link between the language in which pupils take the Programme for International Student Assessment (PISA) test and the scores they achieve in this assessment, focusing upon the case of Wales. Using five rounds of PISA data and an instrumental variable approach, we show how pupils who took the test in Welsh score around 0.3 standard deviations (30 PISA test points) lower in reading, mathematics and science than their peers who took the test in English. This finding is robust to different model specifications and statistical approaches. We argue that this may indicate that the academic performance of teenagers in Wales may be underestimated in PISA – particularly amongst those who take this test in Welsh.

Finland, A Package Deal: Disciplinary Climate in Science Classes, Science Dispositions and Science Literacy

Finland’s educational prowess, though tempered by recent international assessments, has remained intact. This report focused on lessons that could be learned regarding secondary-level science education from the Program for International Student Assessment (PISA) 2015, science-focused assessment. That PISA iteration included not only science literacy but also students’ science dispositions (epistemology, enjoyment, interest, and self-efficacy) and the schools’ science climate measures (disciplinary climate and teaching support). Due to the hierarchical nature of the PISA data, multilevel models were employed in this Finnish study, involving 5582 students from 167 schools. Science dispositions (as outcome measures) were differently associated with teaching support and disciplinary climate (epistemology with neither; enjoyment and interest, with both). Science literacy (as an outcome measure) was associated with all four science dispositions, whether modeled with each science disposition separately or all four simultaneously. Science literacy was also associated with the disciplinary climate in science classes for all tested models. We concluded that, in the Finnish context, science dispositions and the disciplinary climate were predictive of science literacy. Furthermore, we presented evidence from the literature indicating that these conclusions may well extend to other international contexts.

Mediation effects in the relationship between cultural capital and academic outcomes

Following recent literature, this study focused on the mediating mechanisms through which cultural capital leads to students' higher grades and academic ability. Structural modeling was applied to 2018 CZ_PISA (Programme for International Student Assessment) data and the results showed that students' family highbrow cultural resources and reading interest have indirect positive effects on their reading ability and school grades via non-cognitive skills (i.e., students' aspirations and self-concept in reading). The results also implied that only a negligible part of the relationship between students' cultural capital and school grades is mediated by teacher-student relationships, which partly questions the core idea of Bourdieu's cultural reproduction theory. Still, the study suggests that there could be a significant relationship between student's beaux-arts consumption and their reading skills, however, this association could be rather indirect and mediated by student's educational aspirations and self-concept in reading. Finally, the results did not reveal any substantial differences in the mediating mechanism for male and female or low-status and high-status students.

International Comparative Study of Statistics Learning Trajectories Based on PISA Data on Cognitive Diagnostic Models.

The cognitive diagnosis model is an emerging evaluation theory. The mastery of fine-grained knowledge points of students can be obtained via the cognitive diagnostic model (CDM), which can subsequently describe the learning trajectory. The latter is a description of the learning progress of students in a specific area, through which teaching and learning can be linked. This research is based on nine statistical items in the Program for International Student Assessment (PISA) 2012 and an analysis of the response data of 30,092 students from 14 countries from four attributes based on CDM. Then, it obtains the learning trajectory of students in statistical knowledge. The study found that Bulgaria, Costa Rica, Peru, Mexico, and Serbia have the same learning trajectories. The learning trajectories of almost 14 countries are as follows: (1) uncertainty, (2) data handling, (3) statistical chart, and (4) average.

An item response tree model with not-all-distinct end nodes for non-response modelling.

The non-response model in Knott etal. (1991, Statistician, 40, 217) can be represented as a tree model with one branch for response/non-response and another branch for correct/incorrect response, and each branch probability is characterized by an item response theory model. In the model, it is assumed that there is only one source of non-responses. However, in questionnaires or educational tests, non-responses might come from different sources, such as test speededness, inability to answer, lack of motivation, and sensitive questions. To better accommodate such more realistic underlying mechanisms, we propose a a tree model with four end nodes, not all distinct, for non-response modelling. The Laplace-approximated maximum likelihood estimation for the proposed model is suggested. The validation of the proposed estimation procedure and the advantage of the proposed model over traditional methods are demonstrated in simulations. For illustration, the methodologies are applied to data from the 2012 Programme for International Student Assessment (PISA). The analysis shows that the proposed tree model has a better fit to PISA data than other existing models, providing a useful tool to distinguish the sources of non-responses.

What School Factors are Associated with the Success of Socio-Economically Disadvantaged Students? An Empirical Investigation Using PISA Data

Many school-level policies, such as school funding formulae and teacher allocation mechanisms, aim at reducing the influence of students’ low socio-economic condition on academic achievement. Benchmarks and indicators based on large-scale international assessments can be used to measure academic success and identify if and when disadvantaged students are successful. We build on such work and develop a new method for identifying a cross-country comparable metric of the academic success of socio-economically disadvantaged students using data from the Programme for International Student Assessment (PISA). We estimate the prevalence of successful disadvantaged students in 56 countries, as well as changes over time between 2006 and 2015. In addition, we focus on the PISA 2015 edition and explore school factors associated with the probability that disadvantaged students will be successful academically in a subsample of 18 countries. Findings reveal that successful disadvantaged students attend schools with a better disciplinary climate and that provide additional time for instruction in key subjects.